Bayesian Integration of Rule Models

Although Bayesian model averaging (BMA) is in principle the optimal method for combining learned models, it has received relatively little attention in the machine learning literature. This article describes an extensive empirical study of the application of BMA to rule induction. BMA is applied to a variety of tasks and compared with more ad hoc alternatives like bagging. In each case, BMA typically leads to higher error rates than the ad hoc alternative. This is found to be due to the exponential sensitivity of the likelihood to small variations in the sample, leading to eeectively very little averaging being performed even when all models have similar error rates. Coupled with the generation of many models, this causes BMA to have a strong tendency to overrt. An attempt to combat this problem using carefully-designed priors is described. These and further experiments suggest that methods like bagging succeed not because they approximate the optimal BMA procedure better than a single model, but because they eeectively change the prior distribution, compensating for inappropriate simplicity biases in the single-model induction process.